Match recorded violations in strict packs (Part A) - #43
Conversation
|
@technicalpickles @dduugg @martinemde I know it has only been 3 days, but I was hoping to get a quick turnaround on this one which unblocks a spike on my current sprint at work 🙏 If we can't get this merged and released in a timely fashion, I completely understand, please just le me know so I can work off a fork |
dduugg
left a comment
There was a problem hiding this comment.
Thanks for the clear writeup, and for flagging up front that Part B is separable.
Part A is a real bug and the diagnosis looks right to me. package_todo.yml has no field for strict, so Pack::all_violations() hardcoding strict: false (pack.rs:195) isn't something to fix upstream; the asymmetry comes from the file format. I checked every site in src/ that compares or hashes a ViolationIdentifier, and all three are covered. I also confirmed the effect: main prints "There were stale violations found" on uses_strict_mode, this branch doesn't.
Part B matches packwerk on check. I checked the source rather than going off the PR description:
# lib/packwerk/offense_collection.rb
def unlisted_strict_mode_violations
strict_mode_violations.reject { |offense| already_listed?(offense) }
endcheck_command.rb uses that for both display and exit status, so your reading of Shopify/packwerk#368 is accurate.
The mixed case works too, which is the behavior that makes strict mode adoptable: I added a second, unrecorded reference next to the recorded ::Bar and only the new one gets reported.
I left inline notes on the specifics. One blocking issue on update (see the comment on build_strict_mode_violations), plus the docs and release items below.
Blocking (docs): CHECKERS.md says the opposite
CHECKERS.md:16-18:
Setting
enforce_privacytostrictwill forbid all references to private constants in your package. This includes violations that have been added to other packages'package_todo.ymlfiles.Note: You will need to remove all existing privacy violations before setting
enforce_privacytostrict.
Both sentences become false under Part B and need rewriting in whichever PR carries it. CHECKERS.md:101-107 also presents strict_privacy_ignored_patterns as the way to "activate 'strict' mode on your package but have a few privacy violations you know you will deal with later." Part B now covers that case by default, so the docs should say when you'd reach for each.
Should-fix
update's summary message is wrong now. checker.rs:334-348 filters on .identifier.strict with no recorded filter, so in my run update printed "These violations must be fixed for check to succeed" for 2 violations while check said No violations detected!. It had also just deleted the record that made the message false. packwerk uses unlisted_strict_mode_violations for the equivalent message; same filter applies here.
CHANGELOG entry for Part B, in the respect_gitignore who's-affected / what-changes / opt-out format. Same class of change: silent, no config needed to trigger it, different results from the same tree. Two mechanical things: ## Unreleased is stale, since 2fe98b7 is an ancestor of v0.4.0 and everything under that heading already shipped, and at 0.4.0 pre-1.0 a breaking change wants 0.5.0.
Nit
pks update exits 0 where packwerk's update-todo exits 1 when unlisted strict violations exist. Pre-existing and separate from this PR.
Suggestion: take Part A now, split Part B
Part A is a straightforward bug fix, independently useful, and it covers most of what's hurting you (135 stale todos to 0). Part B still needs the write_violations_to_disk preservation fix, the corrected update message, and the CHECKERS.md and CHANGELOG updates, and Part A has to land first for the preservation fix anyway. Given your timeline, splitting probably gets you unblocked sooner than working through Part B here.
One thing to watch when you split: uses_strict_mode is the fixture whose meaning changes, and Part A on its own still changes its output. check should exit 1 with just the two strict messages, no "stale violations" line and no new-violation report. So test_check_with_strict_mode still needs an update in the Part A PR, just a different one than here.
Part B's follow-up would then carry: recorded-strict preservation in write_violations_to_disk, the update message filter, the CHECKERS.md rewrite, the CHANGELOG entry, a test for the listed-strict case on update, and a check -> update -> check test on uses_strict_mode.
On gating Part B behind a config option, I'd say don't. packwerk made it the default with no opt-out, --ignore-recorded-violations already covers the escape hatch, and you've wired it through build_strict_mode_violations the same way build_reportable_violations does it. A pks-only knob would cut against the parity goal.
Verification
cargo test, cargo clippy --all-targets --all-features -- -Dwarnings, and cargo fmt --all -- --check pass on this branch. One unrelated failure, test_gitignore_negation_patterns, reproduces the same way on main (local global gitignore with *.log).
| .violations | ||
| .iter() | ||
| .filter(|v| v.identifier.strict) | ||
| .filter(|v| { |
There was a problem hiding this comment.
Blocking: pks update erases the entries this now depends on.
write_violations_to_disk drops every strict violation when regenerating todo files:
// src/packs/package_todo.rs:144
if violation.identifier.strict {
continue;
}That line is pre-existing and untouched here, but this filter is what starts depending on those todo entries. On main the asymmetry was invisible, because check failed either way. Now, on tests/fixtures/uses_strict_mode, with no source change in between:
$ pks check
No violations detected! # exit 0, working as intended
$ pks update
2 strict mode violation(s) detected. These violations must be fixed for `check` to succeed.
Successfully updated package_todo.yml files! # exit 0
# packs/foo/package_todo.yml is now DELETED. Both of that pack's recorded
# violations are strict, so nothing is written for foo and the None branch
# hits delete_package_todo_from_disk.
$ pks check
2 violation(s) detected: ...
packs/foo cannot have privacy violations on packs/bar because strict mode is enabled ...
# exit 1
A routine pks update un-grandfathers every recorded violation in a strict pack and turns a green build red. Your real-app numbers hold until someone runs update.
I'd call this blocking rather than a pre-existing quirk to port later, because packwerk does the opposite here:
# lib/packwerk/offense_collection.rb#add_offense
if strict_mode_violation?(offense)
add_to_package_todo(offense) if already_listed
strict_mode_violations << offense
else
add_to_package_todo(offense)
endAn unlisted strict violation never gets added, so you can't silence strict mode by running update-todo. An already-listed one gets re-added, and that re-add is what keeps the entry in the file, since PackageTodo#dump writes new_entries wholesale. packwerk protects the state its own check tolerance reads. pks treats both cases the same.
Suggested fix: in write_violations_to_disk, drop only the unlisted strict violations. The recorded set is already at configuration.pack_set.all_violations, the same source CheckAllBuilder uses, and the comparison needs recorded_key(), so Part A comes first.
This doesn't require changing tests/update_test.rs:199-225. That test runs against contains_strict_violations, which ships no package_todo.yml and gets remove_file'd first, so its violation is unlisted, and "todo should not be created for strict violations" is what packwerk does in that case. The already-listed case has no test, which is how this stayed hidden.
| /// it is, and `package_todo.yml` has nowhere to record it, so recorded | ||
| /// violations are always rebuilt with `strict: false`. Compare through this | ||
| /// so a violation in a strict pack can still match its recorded entry. | ||
| pub fn recorded_key(&self) -> Self { |
There was a problem hiding this comment.
Design note, non-blocking, and fine to defer to a follow-up.
Consider moving strict off ViolationIdentifier and onto Violation instead of normalizing at comparison time. Your comment here already says why: strict describes how a violation should be treated, not which violation it is. The doc comment just below at checker.rs:55-64 sets the same rule for source_location, that the identifier defines sameness for comparison against package_todo.yml, "which doesn't store line/column." strict isn't stored there either.
The change is mechanical. Every reader of .identifier.strict (json.rs:56,90; csv.rs:12,53; package_todo.rs:144) already has a full &Violation, and build_strict_violation_message never reads the field. Constructors are pack.rs:195, which is where #41 starts and which then stops having to invent strict: false, plus pack_checker.rs:180 and four test constructors. You'd get all three comparison sites back to plain contains(&v.identifier), #41 becomes impossible to express instead of something a future call site has to remember to guard, and the extra allocations go away.
One alternative to skip: excluding strict from a manual PartialEq/Hash. Violation's derived Eq/Hash delegate to the identifier, and get_all_violations dedupes into a HashSet<Violation>, so making strict: true equal strict: false lets an insert keep the wrong flag, which build_strict_mode_violations then filters on.
recorded_key() is correct as written. This is about where the field lives, not about a bug.
| if violation_path_exists { | ||
| !found_violation_identifiers.contains(todo_violation_identifier) | ||
| !found_violation_identifiers | ||
| .contains(&todo_violation_identifier.recorded_key()) |
There was a problem hiding this comment.
Nit: this call does nothing. todo_violation_identifier comes from pack_set.all_violations, which is always built with strict: false (pack.rs:195), so recorded_key() clones 4 Strings per recorded violation and changes nothing. I reverted just this call and the whole suite stays green.
The found-side .map(|v| v.identifier.recorded_key()) above is the one doing the work. Either drop this one or add a comment saying recorded identifiers arrive already normalized, so a future reader doesn't assume it matters.
| recorded_violations: &'a HashSet<ViolationIdentifier>, | ||
| ) -> anyhow::Result<Vec<&'a ViolationIdentifier>> { | ||
| let found_violation_identifiers: HashSet<&ViolationIdentifier> = self | ||
| let found_violation_identifiers: HashSet<ViolationIdentifier> = self |
There was a problem hiding this comment.
Minor: this moves from HashSet<&ViolationIdentifier> to an owned HashSet<ViolationIdentifier>, so it now clones 4 Strings per found violation rather than copying a pointer. Small next to parsing 15.6k files, so fine to leave.
If you want the cheaper version, a borrowed key tuple that excludes strict avoids the allocations entirely. Moving strict onto Violation (see my note on recorded_key) would also let this go back to borrowing.
| .arg("check") | ||
| .assert() | ||
| .code(0) | ||
| .stdout(predicate::str::contains("No violations detected!")); |
There was a problem hiding this comment.
Test gap worth pinning: a strict pack with some recorded and some unrecorded violations in the same run.
I checked and the behavior is right. Adding a second, unrecorded reference alongside the recorded ::Bar in this fixture reports only the new one. That's the case that makes strict mode adoptable, and nothing in the suite covers it today, so a regression here would be silent.
Also worth a check -> update -> check test on this fixture, asserting the second check is still clean. That's the round trip that currently breaks (see my comment on build_strict_mode_violations).
| .stdout(predicate::str::contains("Violation,Strict?,File,Constant,Referencing Pack,Defining Pack,Message")) | ||
| .stdout(predicate::str::contains("privacy,true,packs/foo/app/services/foo.rb,::Bar,packs/foo,packs/bar,packs/foo cannot have privacy violations on packs/bar because strict mode is enabled for privacy violations in the enforcing pack\'s package.yml file")) | ||
| .stdout(predicate::str::contains( | ||
| "privacy,true,packs/foo/app/services/foo.rb,::Bar,packs/foo,packs/bar,packs/foo cannot have privacy violations on packs/bar because strict mode is enabled for privacy violations in the enforcing pack\'s package.yml file", |
There was a problem hiding this comment.
No coverage lost here, just flagging why for the record: the removed line is byte-identical to the one kept below it, so this drops a duplicate assertion.
The duplication was pointing at something real, though. Unrecorded strict violations get reported twice, since build_reportable_violations doesn't filter on .strict and the formatters concatenate both sets, which is why -o csv emits the same row twice on this fixture. It reproduces on main, so it's pre-existing and not yours to fix here.
87ed033 to
9c16b76
Compare
Fixes rubyatscale#41. `ViolationIdentifier` carries `strict`, but violations rebuilt from `package_todo.yml` always get `strict: false`, so in a strict pack a found violation could never equal its recorded entry. Both comparisons against the recorded set now normalize the found side through `recorded_key()`, which zeroes the flag. That fixes both symptoms: a recorded violation in a strict pack was reported as new, and its todo entry was reported as stale. The todo side needs no normalization. `is_stale_violation` takes its argument from `pack_set.all_violations`, which already rebuilds every recorded violation with `strict: false`, so it is its own recorded key. There is a doc comment saying so, because the asymmetry with the found side reads like an oversight otherwise. `test_check_with_strict_mode` pins the corrected output on `uses_strict_mode`: still exit 1, because strict mode itself is unchanged here, but neither a new-violation report nor a stale-todo line. It fails without the fix.
9c16b76 to
e198118
Compare
Builds on rubyatscale#43, which has to land first: every comparison here needs that PR's `recorded_key()`. `build_strict_mode_violations` now skips violations already recorded in a `package_todo.yml`, matching packwerk's `unlisted_strict_mode_violations` (Shopify/packwerk#368). Turning strict mode on therefore blocks new violations without also requiring the existing list to be emptied first. `--ignore-recorded-violations` still surfaces everything the todo files are grandfathering. Three things had to come with it, because the tolerance reads state that nothing was previously protecting. `write_violations_to_disk` preserved nothing for strict packs: it dropped every strict violation when regenerating todo files, so a routine `pks update` deleted the entries `check` had just started depending on. On `uses_strict_mode` that was check clean, update, check red, with no source change in between. It now drops only the *unlisted* strict violations, so `update` still cannot be used to silence strict mode, but it stops un-grandfathering what strict mode is now tolerating. packwerk keeps the entry for the same reason, in `OffenseCollection#add_offense`. `update`'s summary message filtered on `.identifier.strict` with no recorded filter, so it announced that N violations "must be fixed for `check` to succeed" while `check` reported none. It uses the same filter as the checker now. `CHECKERS.md` asserted the opposite of this behaviour in two places: that strict mode includes violations recorded in other packages' todo files, and that you must clear existing violations before enabling it. Both are rewritten, and the `strict_privacy_ignored_patterns` section now says when to reach for a path exemption rather than a recorded entry, since the recorded case is covered by default. Tests: - `test_check_with_recorded_strict_mode_violation` — the recorded case is clean - `test_check_with_recorded_strict_mode_violation_ignoring_todo` — the escape hatch still reports it - `test_check_with_unrecorded_strict_mode_violation` — an unrecorded strict violation still fails - `test_check_with_partially_recorded_strict_mode_violations` — one recorded and one unrecorded in the same strict pack, in one run. Only the unrecorded one is reported. This is the case that makes strict mode adoptable and nothing covered it, so a regression here would have been silent - `test_update_preserves_recorded_strict_violations` — the recorded entry survives `update`, and the misleading summary line is gone - `test_check_update_check_round_trip_with_strict_mode` — check, update, check, still clean. This is the round trip that was broken `test_check_with_strict_mode_output_csv` moves to `contains_strict_violations`, which ships no todo file, so it still has output to assert against. The duplicate assertion it carried was byte-identical to the one below it, so dropping it costs no coverage. Two new fixtures rather than edits to `uses_strict_mode`, so the mutating tests cannot race the read-only ones: `uses_strict_mode_partially_recorded` and `uses_strict_mode_round_trip`. The CHANGELOG entry follows the `respect_gitignore` who's-affected format. Its `## Unreleased` heading was stale — `2fe98b7` is an ancestor of v0.4.0, so everything under it had already shipped — so that section is now `## 0.4.0` and this change sits under a fresh `## Unreleased`. Pre-1.0, a breaking change like this wants 0.5.0 rather than 0.4.x.
|
Split this per your suggestion, so #43 is now Part A on its own and Part B is up at #45. Description here is rewritten to match, and #45 is rebased on this branch. Part B carries the Thanks for the review, it was unusually thorough. The |
Fixes #41.
ViolationIdentifiercarriesstrict, but violations rebuilt frompackage_todo.ymlalways getstrict: false, so in a strict pack a found violation could never equal its recorded entry.Both comparisons against the recorded set now normalize the found side through
recorded_key(), which zeroes the flag. That fixes both symptoms: a recorded violation in a strict pack was reported as new, and its todo entry was reported as stale.The todo side needs no normalization.
is_stale_violationtakes its argument frompack_set.all_violations, which already rebuilds every recorded violation withstrict: false, so it is its own recorded key. There is a doc comment saying so, because the asymmetry with the found side reads like an oversight otherwise.Now Part A only
@dduugg suggested splitting this and I have taken that. The policy change, which makes
checktolerate recorded strict violations, has moved to #45 together with thewrite_violations_to_diskpreservation fix, theupdatemessage filter, theCHECKERS.mdrewrite, the CHANGELOG entry and the extra tests.Changes from your review
recorded_key()call inis_stale_violation. You were right that it does nothing, sincetodo_violation_identifieris already built withstrict: false. There is a comment there now so a future reader does not assume it matters.test_check_with_strict_modeis updated for Part A rather than for the old combined behaviour, as you described.uses_strict_modestill exits 1 from strict mode, but reports neither a new violation nor a stale todo. It fails without the source fix, which I checked by reverting the source and keeping the test.On the real-app numbers
The 185 strict violations and 135 stale todos in the original description were measured on the combined change, so I am not restating them for this PR. Part A is the half that stops the double reporting, as new and as stale. The strict-violation count needs #45.
cargo test,cargo clippy --all-targets --all-features -- -Dwarningsandcargo fmt --all -- --checkall pass.